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Abstract — Two human-in-the-loop simulation experiments were 
conducted in coordinated fashion to investigate the allocation of 
separation assurance functions between ground and air and 
between humans and automation. The experiments modeled a 
mixed-operations concept in which aircraft receiving ground- 
based separation services shared the airspace with aircraft 
providing their own separation service (i.e., self-separation). 
Ground-based separation was provided by air traffic controllers 
without automation tools, with tools, or by ground-based 
automation with controllers in a managing role. Airborne self- 
separation was provided by airline pilots using self-separation 
automation enabled by airborne surveillance technology. 

The two experiments, one pilot-focused and the other controller- 
focused, addressed selected key issues of mixed operations, 
assuming the starting point of current-day operations and 
modeling an emergence of NextGen technologies and procedures. 
In the controller-focused experiment, the impact of mixed 
operations on controller performance was assessed at four stages 
of NextGen implementation. In the pilot-focused experiment, the 
limits to which pilots with automation tools could take full 
responsibility for separation from ground-controlled aircraft 
were tested. 

Results indicate that the presence of self-separating aircraft had 
little impact on the controllers’ ability to provide separation 
services for ground-controlled aircraft. Overall performance was 
best in the most automated environment in which all aircraft 
were data communications equipped, ground-based separation 
was highly automated, and self-separating aircraft had access to 
trajectory intent information for all aircraft. In this environment, 
safe, efficient, and highly acceptable operations could be achieved 
for twice today’s peak airspace throughput. In less automated 
environments, reduced trajectory intent exchange and manual 
air traffic control limited the safely achievable airspace 
throughput and negatively impacted the maneuver efficiency of 
self-separating aircraft through high-density airspace. In a test 
of scripted conflicts with ground-managed aircraft, flight crews 
of self-separating aircraft prevented separation loss in all 
conflicts with detection time greater than one minute. In debrief, 
pilots indicated a preference for at least five minute’s alerting 
notice and trajectory intent information on all aircraft. When 
intent information on ground-managed aircraft was available, 
self-separating aircraft benefited from fewer conflict alerts and 
fewer required deviations from trajectory-based operations. 

Keywords - air traffic management, function allocation, separation 
assurance, automation, self-separation, air/ground integration 


I. Introduction 

An essential part of developing the Next Generation Air 
Transportation System (NextGen) is the exploration of new 
technologies, procedures, and human roles in providing 
services and functions for the safe and expeditious passage of 
aircraft. Separation assurance is a key function of Air Traffic 
Control (ATC) and a core responsibility of air traffic 
controllers in current-day operations. Due to the safety 
criticality of separation assurance, a complex system of 
airspace and route structures, surveillance and communication 
technologies, and operational controls on aircraft trajectories 
has evolved to enable a separation assurance environment in 
which controllers, using voice communication with pilots, can 
sustain safe operations with manageable workload. 

This evolved, complex system is reaching its limits in 
accommodating new demand and satisfying operator needs for 
efficiency. A significant constraining factor is the controller’s 
workload in communicating with and separating aircraft. 
Currently, the controller is responsible for nearly all separation- 
related functions. Ground automation plays an ancillary role in 
enhancing controller situation awareness, and the aircraft (i.e., 
the flight crew and airborne automation) has a passive role with 
respect to separation, simply obeying trajectory instructions 
(except in collision avoidance situations where an active role is 
taken). Given that human workload capacity cannot be 
substantially increased, other means will be needed to stretch 
beyond the current limits. 

The Concepts and Technology Development Project of the 
NASA Airspace Systems Program is exploring fundamental 
changes to separation assurance that reduce or eliminate human 
workload as a limiting factor. Through this “function 
allocation” research thrust, NASA researchers are testing 
separation concepts that leverage the extensive use of 
automation and the untapped, distributed resource of aircraft 
systems and crews. Concepts for ground-based and airborne 
separation developed and researched over the last decade and 
beyond are brought together into a “mixed operations” 
environment where the maximum use of all resources for 
separation can be explored. 

The focus of this mixed-operations research activity is non- 
segregated airspace where airborne and ground-based 
separation capabilities coexist. If it can be made sufficiently 
safe, integrated mixed operations provides more flexibility to 



aircraft operators than homogeneous or segregated operations. 
It allows the aircraft operators to choose the method of 
trajectory management, ground-based or airborne, that is the 
most cost-effective for each flight without compromising their 
access to the most efficient routes and altitudes. Having this 
flexibility is economically valuable to the operator community, 
given the large variations in operator business models, aircraft 
equipment, stage lengths, and flight-optimization objectives. 
Even though ground-based and airborne concepts have each 
shown potential for scaling and efficiency, a multi-option 
concept provides more opportunity for achieving these goals 
and less implementation risk. By developing mixed-operations 
concepts, NASA’s goal is to provide the largest range of viable 
solutions for the aviation community. 

Several key questions were identified in the context of 
enabling mixed operations (those for which separation of traffic 
in shared airspace is managed through multiple function- 
allocation means). These questions form the basis of the two 
coordinated simulation experiments (pilot-focused and 
controller-focused) . 

A. How does the presence of self-separating aircraft affect the 

performance of the ground-based separation system? 

Since any new separation system will not be built from 
scratch but rather will build upon the existing, complex system 
of skilled controllers and well-established procedures, it is 
important that the performance of the existing system not be 
compromised in the process of achieving the system with 
newly allocated separation functions. To address this question, 
the controller-focused experiment was designed to assess 
ground-based separation performance in traffic environments 
with and without self-separating aircraft. 

B. What are the limits of ability of self-separating aircraft to 

shoulder the burden of mixed operations? 

The complexity of the existing ground-based system may 
preclude making significant changes to accommodate the 
introduction of new operations such as self-separation. 
Therefore, the most feasible approach to implementation is for 
the self-separating aircraft to give way to ground-controlled 
aircraft (at least in the early stages of implementation) and to 
avoid placing new operational restrictions on the ground-based 
control of traffic. In practical terms, the controller should be 
free to focus almost exclusively on ground-controlled traffic, 
while self-separating aircraft must be capable of resolving all 
conflicts with ground-controlled aircraft (in addition to 
conflicts with other self- separating aircraft), regardless of 
warning time or encounter geometry. There will be limits to 
this ability, and thus the pilot-focused experiment was designed 
to assess self-separation performance in a variety of (primarily 
short-notice) conflicts with ground-managed aircraft. 

C. How will the implementation and maturation of NextGen 

affect the ability to operate with mixed operations? 

Envisioned as part of NextGen are new technologies such 
as air/ground data communications (data comm.) and decision- 
support automation for controllers and pilots. As NextGen 
matures, the number of data comm, equipped aircraft is 
expected to grow, enabling greater controller reliance on 
automation for managing trajectories (and therefore managing 


separation). NextGen technologies will also support greater 
air/ground sharing of trajectory intent information. The 
question to be addressed is whether the concept for mixed 
operations (i.e., the means by which self-separating and 
ground-managed aircraft share the airspace) is compatible with 
the expected emergence of NextGen. To address this question, 
the controller-focused experiment tested mixed operations at 
different stages of the NextGen evolution of ground-based 
automation (i.e., over a range of minimum to maximum 
envisioned capabilities), and the pilot-focused experiment 
tested NextGen self-separation operations with and without 
access to the trajectory intent of ground-managed aircraft. 

The two human-in-the-loop simulation experiments were 
designed and conducted in a coordinated effort to explore these 
questions that arise in function allocation of separation 
assurance. This mixed operations research builds upon a recent 
separate comparison study of each concept operating separately 
[1]. Of the preceding set of key questions, Question A was 
addressed in a controller-focused experiment conducted in the 
Airspace Operations Lab (AOL) at NASA Ames Research 
Center. Question B was addressed in a pilot-focused 

experiment conducted in the Air Traffic Operations Lab 
(ATOL) at NASA Langley Research Center. Different 
elements of Question C were addressed in both experiments. 
This paper presents a summary of these experiments and their 
top-level results. Section II describes separation assurance 
concepts (airborne and ground-based) and their mixed- 
operations integration. Sections III and IV respectively 
describe the controller-focused and pilot-focused experiments 
and present initial results. Section V gives conclusions. 

II. Separation Function Allocation Concepts 

A. Airborne Separation Concept 

The airborne separation concept leverages the attributes of 
both distribution and automation in its approach to function 
allocation. In this approach, separation functions for individual 
aircraft are performed onboard the aircraft (i.e., the “ownship”) 
to provide separation from all traffic the ownship encounters. 
The aircraft (rather than the controller) manages its own 
trajectory during en route flight, and it “self-separates” from all 
traffic by detecting conflicts and adjusting its trajectory as 
needed. With multiple self-separating aircraft in the airspace, 
the separation “service” is distributed among them and resident 
onboard each equipped aircraft. Separation automation 
onboard the aircraft is heavily leveraged to avoid the flight 
crew having to provide such capability manually. 

Aircraft that manage their own trajectory and separation are 
referred to as flying under Autonomous Flight Rules (AFR). 
AFR distinguishes these aircraft from IFR aircraft, which are 
managed by and receive separation services from ATC. While 
AFR aircraft optimize their own trajectory through airspace 
shared with IFR traffic, the mixed operations concept tested in 
these experiments requires AFR aircraft to yield right-of-way 
to IFR aircraft in all conflict encounters and to take 
responsibility for ensuring the separation standard is met. To 
meet this responsibility, the flight crew uses onboard 
automation that processes data from ownship avionics and 
airborne surveillance (Automatic Dependent Surveillance 
Broadcast, ADS-B) to probe for conflicts and compute 



resolution maneuvers, and possibly several acceptable 
alternatives. The crew chooses the desired maneuver and 
executes it directly. Because the separation function is 
performed onboard, no ATC approval is needed to maneuver. 
AFR intent information is electronically available to 
controllers, but they bear no responsibility for separation 
between AFR and IFR aircraft. Coordination between AFR 
pilots and controllers, if needed, is conducted by voice 
communication. 

NASA has developed and investigated the AFR concept for 
over a decade, designing prototype flight-deck automation and 
conducting numerous analyses, batch simulations, and human- 
in-the-loop experiments. A more thorough description of the 
AFR concept is provided in [2]. 

B. Ground-Based Separation Concept 

Separation assurance in the National Airspace System today 
is ground-based, manual, and limited by how many aircraft the 
air traffic controller can keep under positive control. Emergent 
technologies are intended to support air traffic controllers in 
detecting conflicts and generating solutions and will reduce 
some of the coordination and communication workload. The 
current near- and mid-term NextGen plans foresee the 
introduction of additional decision support tools to improve 
operations, but under the current paradigm the human operator 
remains responsible for providing separation between all 
aircraft. [3] 

In 2000 Erzberger [4] introduced the automated airspace 
concept, which was conceptualized as a completely automated 
ground-based separation assurance concept with two layers: a 
trajectory-based layer providing efficient resolutions for non- 
urgent conflicts to be communicated to the flight deck’s flight 
management system via data communication, and a tactical 
layer providing short term conflict avoidance maneuvers to 
keep aircraft separated in case the trajectory-based layer failed. 
Human- Systems Integration research at NASA Ames has taken 
components of this fully automated concept, integrated them 
into transitional stages and developed a ground-based concept 
in which humans and automation collaborate to substantially 
increase airspace capacity while retaining acceptable operations 
for the operators. [5] 

This concept of ground-based automated separation 
assurance utilizes technologies to shift the workload-intensive 
tasks of monitoring and separating traffic from the controller to 
the automation. A critical element of this centralized concept 
makes the ground-side automation, not the controller 
responsible for conflict detection. The automation is also 
responsible for monitoring the compliance status of all aircraft 
relative to their reference trajectory. In many cases, the 
automation, not the controller, is responsible for resolving 
conflicts as well. However, the controller is responsible for 
maintaining separation of unequipped aircraft using a 
conventional voice link and steps in to handle certain off- 
nominal situations. Thus, under automated separation 
assurance, air traffic controllers’ roles involve providing 
services and performing decision-making activities, while the 
roles of monitoring, providing nominal separation, and back-up 
solutions in off-nominal situations are allocated to the 
automation. 


III. Controller-focused Experiment 

The controller- focused experiment addressed the impact of 
self- separating aircraft on ground-based separation assurance 
services in different stages of NextGen. Conducted at the Ames 
AOL, the experiment compared the air traffic controllers’ 
ability to provide separation services in increasingly advanced 
operational environments with and without the presence of self- 
separating aircraft. The primary questions with regard to mixed 
operations were (1) whether the self- separating aircraft would 
impact the controller’s (and ground-based automation’s) ability 
to separate conventional aircraft (Question A in Section I) and 
(2) how these operations would unfold under different 
emergent stages of NextGen (Question C in Section I). 

In order to investigate the first question, multiple stages of 
NextGen were run in both IFR-only and mixed IFR/ AFR 
scenarios. The only difference between the IFR-only and the 
mixed-operations scenarios was that eight selected aircraft, 
operating according to IFR rules in one case, were operated 
under AFR rules in the corresponding mixed-operations case. 
The remaining aircraft were all operating under IFR rules. 

In order to investigate the second research question, four 
different NextGen stages were run. The first stage, “Baseline”, 
was designed to provide data approximating current day 
operations with the addition of ADS-B out surveillance data. 
The mixed-operations runs were designed to indicate what 
would happen if aircraft conducted self-separation in a very 
near-term environment that is characterized by voice 
communication and manual control. Controllers had to verbally 
issue frequency changes to the AFR aircraft but had no 
separation responsibility for them. AFR aircraft and controllers 
had access to ADS-B state data for all traffic, but no ADS-B 
intent information. The traffic levels were selected to be 
representative of current day peak traffic levels with a Monitor 
Alert Parameter (MAP) value of 1 8 aircraft per sector. 

The second stage, labeled “Minimum NextGen”, introduced 
limited data communication between the ground-side and 25% 
of the simulated aircraft. This data communication enabled an 
automatic transfer of communication of aircraft from one sector 
to the next. This eased the controller workload in handling 
those aircraft, which included the AFR aircraft under mixed 
operations. It was expected that controllers could potentially 
ignore the AFR aircraft, because they had no routine duties 
with regard to them. This stage also introduced more decision 
support capabilities for the controllers, none of which were 
integrated with data comm. So, all control instructions still had 
to be communicated via voice. It was hoped that the new 
technologies could enable a capacity increase of 20%, and the 
MAP value was set to 22 aircraft per sector for the “Minimum 
NextGen”. 

In the third stage, entitled “Moderate NextGen”, the 
controller planning tools and the flight management systems 
on-board the aircraft were integrated with data comm., and 
50% of the aircraft were assumed data comm, equipped. Unlike 
the previous two stages, AFR aircraft in this condition had 
access to trajectory intent of up to four trajectory change points 
for all other aircraft. Controllers were able to issue trajectory 
change instructions to equipped aircraft via data comm. Based 
upon earlier research, it was hypothesized that this environment 



could enable a capacity increase of 50% over the Baseline and 
therefore the MAP value was set to 27 for this stage. 

In the final NextGen stage, referred to as “Maximum 
NextGen”, the responsibility for conflict detection among IFR 
aircraft was assigned to the ground automation and all aircraft 
were data comm, equipped. When conflicts were detected, 
ground automation computed trajectory-based resolutions and 
issued those directly to the flight deck, as long as the computed 
resolutions did not violate preset tolerances. Otherwise, the 
conflict was flagged to the controller for resolution. Prior 
research had indicated the scalability of this approach, and 
therefore the traffic levels were selected at 100% over the 
Baseline with a MAP value of 36 aircraft per sector. 

A. Experiment Design 

The experiment used a within- subjects design with two 
air/ground function allocation concepts (IFR-only and mixed 
IFR/AFR) over four stages of NextGen, resulting in eight 
experimental conditions. Prior tests had shown that the 
complex nature of changes between the NextGen stages made 
it impossible for participants to operate within the different 
environments in a randomized or counter-balanced order. 
Therefore, the conditions were run in order of their 
maturation/automation level: first Baseline, then Minimum, 
then Moderate, and finally Maximum NextGen. In each stage, 
the pilots and controllers were briefed and trained for one day 
on the operational environment, tools, and procedures, 
followed by one day of data collection in which they ran a total 
of six scenarios alternating between IFR and mixed IFR/AFR 
operations. The test matrix is shown in Table 1. 

The test airspace consisted of five simulated high altitude 
sectors in Cleveland Center (ZOB) that were combined into 
two air traffic control areas (north and south), controlling all 
traffic at FL 320 and above (Fig. 1). Six current and one very 
recently retired Federal Aviation Administration (FA A) front 
line managers served as primary participants, five as radar 
controllers and two as area supervisors. Five retired controllers 
operated radar associate positions in the test airspace, and 
three retired controllers controlled the traffic in the “ghost” 
sector outside the test airspace. Ten airline pilots operated 
eight mid-fidelity, single-aircraft flight simulators, and ten 
general aviation/corporate pilots operated multi-aircraft 
stations. 

B. Facility and Test Scenario Description 

For the experiment, the AOL at Ames [6] was configured 
with two air traffic control rooms hosting the five test sectors. 


Table 1 : Test Matrix for controller focused experiment 
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Figure 1. Test airspace for controller focused experiment. 


Figure 2. AOL sector position. 


Each sector was configured 
with a radar display and a 
nearly identical radar 
associate display (Fig. 2). 

A supervisor participant 
managed each room and 
brought radar associates on 
position when needed. The 
air traffic control 
environment, controller 
workstations, and majority of IFR aircraft were simulated using 
the Multi Aircraft Control System (MACS) developed at Ames. 
The AFR aircraft in the mixed-operations conditions and their 
respective IFR counterparts in the IFR-only conditions were 
simulated using the higher fidelity ASTOR desktop flight 
simulators developed at Langley and operated by type-rated 
airline pilots. The ASTORs and their capabilities are discussed 
in more detail in the description of the pilot-focused 
experiment. Traffic scenarios consisted of overflights, 
departures, and arrivals into nearby airports originally based on 
actual ZOB traffic flows, but modified for the higher traffic 
densities used in the simulations. 



C. Results of Controller-Focused Experiment 

1) Separation Assurance Performance 
Separation assurance performance was initially analyzed 
by comparing the loss of separation (LOS) events between the 
IFR-only and the mixed IFR/AFR conditions for the different 
stages of NextGen. Table 2 shows a summary of the 
conditions and the combined number of LOS events as well as 
the number of ASTORS involved in these LOS events after 
discounting simulation artifacts, pseudo pilot, and ghost 
controller errors. Each of the data points represents 120 
minutes of simulation time across five test sectors. 


Approximately half of all the LOS events could be traced 
to automation failures, causing late conflict detections, and the 
other half to operator/automation interaction failures, e.g., 


Table 2: LOS events involving all aircraft and ASTORs only 
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ignoring a conflict warning or failing to probe a trajectory 
change before issuing it to the aircraft. There were no 
significant differences between IFR only and mixed operations 
within each NextGen stage. The ASTORs were involved in 
four LOS events, two as IFR in the IFR-only conditions and 
two as AFR in the mixed conditions. 

Clearly, there were differences between the various stages 
of NextGen. The Minimum and especially the Moderate 
NextGen condition resulted at times in unacceptable separation 
assurance performance, because the traffic levels were 
increased beyond the complexities/levels that the operational 
environment could handle. Worst performance overall was 
observed in the Moderate IFR-only condition. Identifying the 
exact factors that made the scenarios unworkable will be the 
subject of further analysis and future publications. Remarkably, 
the Maximum NextGen condition performed very well at twice 
current traffic levels. The subjective data mirror the separation 
assurance performance very well. 

2) Controller Workload 

Every three minutes during a run, participants were asked 
to rate their subjective workload experience in real-time on a 1- 
6 scale from ‘very low’ to ‘very high’. Participants rated their 
workload between 1 and 6 for all but the Maximum condition, 
in which their rating did not exceed 3. Mean ratings and 
standard errors are shown in Fig. 3. On average, participants’ 
rated their workload at just over 3 (“moderate”), except in the 
Maximum condition, where participants’ mean workload rating 
was just below 2 (“low”), i.e., participants reported their 
workload dropped in the Maximum condition significantly. 
However, there was no significant difference in workload 
between IFR/AFR and IFR-only conditions within any of the 
four NextGen conditions. Controllers’ workload experience 
was also verified through post-hoc ratings. The results closely 
reflect the real time workload ratings and will be published 
separately. 

3) Acceptability 

Once for each condition, controllers were asked a series of 
six questions that form the Controller Acceptability Rating 
Scale (CARS) [7]. The CARS scale is comprised of four 
ratings, where participants indicate the safety, controllability, 
and satisfactoriness of the separation assurance operations and 
then rate its acceptability Comparing the CARS ratings over 
the eight conditions, most of the controllers (84%) rated the full 
set of separation assurance operations as “acceptable”, with 
only 16% giving a less than acceptable rating. Fig. 4 shows the 



Figure 3. Mean real-time workload and standard error for all controllers 
under the 8 study conditions 


means for each study condition when these ratings are 
combined. The mean ratings emphasize that the controllers 
rated the SA operations as safe and controllable with some 
amount of compensation on their part to make the operations 
run smoothly, as all the mean ratings fall between 6 
“considerable compensation” and 10 “desired performance 
reached”. Although the mean CARS for mixed IFR/AFR and 
IFR-only conditions within each NextGen condition are very 
similar, whenever there is a difference, the mixed IFR/AFR 
rating is slightly higher than the IFR-only rating. The main 
variation in mean ratings occurs between NextGen conditions. 

While the average CARS rating was high for the Baseline, 
Minimum, and Maximum NextGen conditions at around 9 
“minimal controller compensation” (a little lower for the 
Baseline-IFR condition), it was noticeably lower for the two 
Moderate conditions at 6.75. Although the averages mask it, 
this lower mean is not due to slightly lower ratings from all 
controllers but stems from two participants rating the 
operations as “unsafe”. Their reasons for the unsafe ratings 
were mainly that they were not alerted to conflicts: “close 
vicinity descents were not alerting until after separation would 
have been lost”. And secondly, they indicated that they were 
put in positions where they couldn’t control the traffic in time, 
e.g., “no control over several situations in which aircraft were 
already in dangerous proximity to each other”. 

4) Situation awareness 

An impression of how much controllers felt they 
understood about the new separation assurance automation and 
operations was targeted by using the Situation Awareness 
Rating Tool (SART) [8]. The SART is comprised of 3 
subscales that participants rate on a 7-point scale. These 
ratings are then combined to generate an overall situation 
awareness (SA) rating for each participant in each run. With 7- 
point sub-scales, the SART scores range from -5 “very low 
SA” to 13 “very high SA”; a score of 4 is “medium SA”. 

Fig. 5 shows that, on average, in the Baseline and Moderate 
NextGen conditions, participants rated their awareness at just 
above “medium SA” (4). Ratings in the Minimum NextGen 
condition were slightly higher, and in the Maximum NextGen 
condition, participants’ scores were above 10, indicating they 
felt they had “high” awareness. This is complementary to the 
workload ratings and indicates that controllers perceived the 
Maximum NextGen condition differently. SART scores were 
compared between mixed IFR/AFR and IFR-only conditions in 
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Figure 4. Mean CARS rating for each study condition 





Figure 5. Mean Situation Awareness and standard error reported by 
controllers under the 8 study conditions 


each of the four NextGen conditions using a Wilcoxon signed 
rank test, but none of the differences between the mixed 
IFR/AFR and IFR-only conditions were significant. 

The SART subcategory ratings indicated that in the 
Maximum conditions controllers said there was very low 
demand in the mixed IFR/AFR condition and low demand in 
the IFR-only condition. Correspondingly, controllers rated that 
they had more spare capacity and slightly higher understanding 
in the Maximum NextGen conditions. 

5) Flight-path efficiency 

In order to gather additional insight into the effectiveness 
of mixed operations under the various stages of NextGen, the 
flight paths of the ASTOR aircraft flying under AFR rules were 
compared to their counterparts flying under IFR rules for all 
NextGen conditions. Since AFR aircraft had to avoid all other 
IFR traffic, they were expected to maneuver more frequently 
than IFR traffic. In order to get a first impression on this 
potential impact of additional resolution maneuvers and how 
NextGen environment differences might affect flight-path 
efficiency in dense air traffic environments, the progress the 
ASTORs made towards their destination was computed as the 
distance to the destination at the beginning of a scenario minus 
the distance after 20 minutes of flight time (chosen because 
some ASTORS exited the test airspace shortly after 20 
minutes). As shown in Fig. 6, for the three NextGen conditions 
excluding Maximum NextGen, aircraft operating under AFR 
exhibited delayed mean progress toward destination relative to 
operating under IFR on the same initial routes, likely a 
reflection of additional flight miles flown. In part, the behavior 
is expected because AFR aircraft always gave way to IFR 
aircraft, whereas under IFR, the aircraft had a 50-50 chance of 
not being selected by ATC to maneuver for conflict resolution. 



Figure 6. Progress towards destination after 20 minutes flight time (mean 
and standard error). 


However, NextGen equipage and information assumptions 
were likely contributing factors as well. In the Baseline and 
Minimum NextGen conditions, AFR aircraft were not given 
access to IFR intent information. In addition, the Baseline to 
Moderate NextGen conditions included some voice-only 
aircraft for which intent information (available in the Moderate 
NextGen condition) was less reliable. The lack of intent 
information may have resulted in additional or prolonged 
tactical maneuvering by AFR aircraft, which can be less 
efficient. In the Maximum NextGen condition, aircraft 
progressed toward their destination equally well under AFR 
and IFR rules, indicating that in a truly trajectory-based 
environment, where intent information is ubiquitously available 
and reliable, both types of operations may have similar 
operating efficiency. 

IV. Pilot-focused Experiment 

The companion experiment in this coordinated research 
study addressed the perspective of the AFR pilot in mixed 
operations. Conducted at the Langley ATOL, the experiment 
focused on two issues: the ability of AFR aircraft to shoulder 
the burden of detecting and resolving conflicts with ground- 
controlled IFR aircraft, and the value of AFR aircraft having 
access to IFR intent information. The experiment was 
organized in two parts, a set of “primary” runs to examine the 
first issue and a set of “exploratory” runs for the second issue. 

The primary portion of the pilot-focused experiment was 
designed to identify the limits under which AFR aircraft can 
ensure separation from IFR aircraft in normal operations. The 
parameters of interest included amount of alerting time and 
conflict geometry. It was hypothesized that achieving adequate 
separation performance requires some minimum amount of 
alerting time for pilots, and that conflict geometry does not 
have an interaction effect with this alerting time. To ensure 
AFR aircraft can shoulder the separation burden of mixed 
operations, it will be important that this minimum required 
alerting time be guaranteed through a combination of ATC 
restrictions on IFR maneuvers, IFR intent information sharing, 
and AFR automation design and parameter settings. An 
example of the latter is the use of extra separation buffers 
beyond the minimum required separation standard, a technique 
initially explored in this experiment. To test the effects of 
alerting time and conflict geometry, a series of scripted 
conflicts with a range of carefully controlled IFR maneuver 
timing and encounter orientation were created for AFR flight 
crews to resolve using automation tools. Controllers did not 
participate in these runs. 

The exploratory portion of the experiment addressed the 
value of AFR aircraft having access to trajectory intent 
information from IFR aircraft. Currently, the ADS-B mandate 
does not require broadcast of intent, either trajectory change 
points or target state information (e.g., target headings and 
altitudes). The absence of this intent information may cause 
conflicts to “pop-up” with shorter notice, as planned IFR 
maneuvers are executed without forewarning. Pop-up conflicts 
can also occur when controllers turn, climb, or descend IFR 
aircraft in previously unplanned maneuvers in proximity to 
AFR aircraft, e.g., to resolve conflicts between IFR aircraft. 
These events can be mitigated through ATC-to-AFR voice 




coordination or through ATC restrictions on IFR maneuvering 
near AFR aircraft, but the parameters of these elements need to 
be defined. To explore the issues surrounding intent 
information exchange, AFR flight crews flew a series of 
scenarios representing normal en route operations with and 
without automatic data transmission of IFR intent information. 
Confederate controllers and IFR aircraft pilots participated in 
these runs to provide for normal interactions. 

A. Experiment Design 
1) Primary Matrix 

To test the ability of AFR pilots to resolve conflicts of 
varying timing and geometry, a test matrix of conflict scenarios 
was generated using a fractional [4]x[2x2x2] between-subjects 
design with four categorical factors, shown in Table 3. The 
first factor was Time to Buffer Loss (TBL), the amount of 
alerting time given to pilots prior to reaching a buffered 
protected zone around the IFR aircraft (8 nmi lateral separation, 
1000 ft vertical separation). Alerting times within each bin 
were assumed identical for analysis purpose. The quantity of 
data points planned for the first four bins was 12, 12, 18, and 
24, respectively, emphasizing the shorter alerting times of 
greater research interest. An additional bin for alerting times 
less than 20 seconds included only vertical encounters, due to 
the difficulty in creating such short-notice lateral encounters. 
This bin contained six data points and was generally not 
combined with the other bins for statistical analysis. 

The 2x2x2 portion of the design represented three elements 
of conflict geometry: encounter angle , maneuver dimension , 
and passage orientation. The first element, encounter angle , 
was divided into convergence angle bins with the two aircraft 
either roughly aligned (“acute”) or roughly opposed (“ obtuse ”). 
Maneuver dimension refers to the IFR aircraft approach 
direction being either “lateral” (same altitude and level with 
the AFR aircraft) or “vertical” (climbing or descending into 
separation loss, with the AFR aircraft always level). Passage 
orientation refers to whether the IFR aircraft in the encounter 
would “pass in front” of the AFR aircraft or “pass behind” if 
the conflict were left unresolved. 

A total of 34 airline pilots participated as 17 flight crews, 
with each crew flying 12 scenarios in the primary matrix. Each 
flight crew saw 12 scripted conflicts, but not the same 12 
conflicts. A between-subjects design with blocking by groups 
of 3 crews was used to provide balanced coverage of the 
fractional test matrix, where not every geometry combination 
could be tested at every TBL condition. A complete 
description of the primary matrix experiment design and 
conditions tested is presented in [9]. 


TABLE 3. Factors included in primary matrix. 


Factors 

Values 

Time to 
Buffer Loss 

10-4 4-2 2-1 60-20 + 

<20 sec. 

min. min. min. sec. 

Encounter 

Angle 

Acute (0-50 deg.) 

Obtuse (130-180 deg.) 

Maneuver 

Dimension 

Lateral 

Vertical 

Passage 

Orientation 

Pass in front 

Pass behind 


2) Exploratory Matrix 

To explore the value of IFR intent information in the 
context of normal mixed-operations procedures, a 2x1 within- 
subjects design was employed, with the one categorical factor 
of IFR intent information availability. With IFR Intent On, 
AFR aircraft received up to four trajectory change points for 
IFR aircraft, whereas with IFR Intent Off, only state 
information was received. Twelve flight crews participated in 
the exploratory runs in groups of six across two weeks. Over 
two weekly sessions, a total of 72 flights were accomplished, 
split evenly between IFR Intent On and IFR Intent Off (i.e., 12 
flight crews each flying three scenarios in each of the two 
intent conditions). The flight duration was 30 minutes. Traffic 
density matched the Moderate NextGen condition (MAP = 27 
aircraft per sector) in the controller-focused experiment, with 
all aircraft flying as IFR except the six subject AFR aircraft. 
Each run included one scripted conflict (TBL > 9 min.) near 
the beginning and additional conflicts occurring naturally 
following the initial resolution maneuver. 


B. Facility and Test Scenario Description 

The Langley ATOL was configured with six “team pilot” 
stations that permitted a flight crew to share a desktop 
simulator. Referred to as an ASTOR, the desktop simulator 
provides the displays and controls of a modern Boeing-style 
widebody jet aircraft. Integrated with the avionics system is an 
automation tool designed to support the flight crew in self- 
separation operations. The Autonomous Operations Planner 
(AOP) [10] provides a full suite of conflict detection, 
resolution, and prevention tools, using information obtained 
from ownship systems (aircraft state, autoflight settings, active 
strategic route, pilot-specified tactical flight targets) and ADS- 
B (traffic aircraft states, intent information if available), among 
other sources of information. AOP supports both “strategic” 
trajectory-based operations, i.e., fully coupled to the Flight 
Management System (FMS), as well as “tactical” operations 
using pilot-specified flight targets set on the Mode Control 
Panel (MCP). The flight crew is alerted to conflicts detected 
by AOP on a textual display, as well as by audible alerting and 
graphical depiction on the 
Navigation Display. Fig. 7 
shows an example AOP 
display for a “tactical urgent” 
conflict. 

In addition to the six 
ASTOR stations serving as 
AFR aircraft, the ATOL was 
configured with six MACS 
stations: two sector controller 
stations for the test sectors, one 
ghost controller station for the 
surrounding airspace, and three 
“pseudo-pilot” stations from 
which the IFR aircraft were 
flown. The MACS controller 
stations were configured in the 
Moderate NextGen mode (as 
described earlier for the 
controller-based experiment) Figure 7. Example AOP display 
with the corresponding of a tactical-urgent conflict. 



t Analyzed separately 



advanced suite of controller automation tools, IFR trajectory 
data comm., and auto-handoff capability for all aircraft. These 
stations were staffed in the exploratory runs. The scripted 
primary runs required no human controllers or pseudo-pilots. 

The primary runs used the same Cleveland Center sectors 
tested in the controller-based study (Fig. 1), and the exploratory 
runs used a two- sector subset. The traffic files and densities 
matched the Moderate NextGen condition. The durations of 
the primary and exploratory runs were 10 and 30 minutes, 
respectively. The short primary runs were sufficient for 
detecting and resolving a single conflict, whereas the longer 
exploratory runs permitted operations to unfold naturally and 
produce more extensive interactions. Each pilot completed a 
survey after each primary and exploratory run. Additional pilot 
comments were captured during group debrief sessions. 

C. Results of Pilot-Focused Experiment 

1) Separation Assurance Performance (Primary Runs) 

The purpose of the primary test matrix was to determine the 
limits of ability of self-separating aircraft to ensure separation 
from ground-managed IFR aircraft as functions of alerting time 
and conflict geometry. A total of 188 of 204 conflict scenarios 
were analyzed, excluding 16 scenarios where flight crews 
maneuvered preemptively and disrupted the scripted conflict. 

Table 4 presents the LOS results. The separation standard 
for analysis was 5 nmi and 800 ft. All conflicts with alerting 
times greater than one minute were resolved without LOS. In 
the 20-60 sec. bin, 11 LOS events occurred out of 62 runs, 
excluding three borderline cases. A two-sided Fisher’s Exact 
Test [11] indicated the difference in proportions of LOS was 
significant between the top four bins (p = 0.016) and between 
the top three combined bins and the 20-60 sec. bin (p = 0.001). 
Thus, the data show that with at least one minute alerting, the 
flight crews using AOP tools and procedures were effective in 
ensuring separation from ground-controlled aircraft. However, 
many pilots indicated in their debrief comments that alerting of 
five or more minutes would likely be necessary for operational 
acceptability. 

Statistical tests were unable to detect an effect of the three 
geometry variables (< encounter angle, maneuver dimension, 
passage orientation ) on the LOS results. Analysis of 
individual LOS scenario recordings confirmed that the primary 
factor in most cases was lack of sufficient time to initiate and 
complete a maneuver. Additional factors in some scenarios 
included crew response delay, desktop simulation control 
issues (mouse vs. real knobs and switches), and a simulation 
bug (uncommanded auto-flight mode switch). Two of the 1 1 
LOS cases were Operational Errors with lateral separation less 


TABLE 4. LOSS OF SEPARATION RESULTS BY INITIAL ALERT TIME. 


Initial Alert Time 

Losses of Separation 

Number of Runs 

4-10 minutes 

0 

33 

2-4 minutes 

0 

32 

1-2 minutes 

0 

46 

20-60 seconds 1 

11 

62 

< 20 seconds 

0 

15 

Total 

11 

188 


than 4.5 nmi. The minimum separation for the smaller of these 
was 3.97 nmi. 

To provide an extra safety margin in AFR-to-IFR 
separation, the primary conflict-detection logic employed a 3 
nmi lateral buffer. Thus, AOP alerted conflicts if the distance 
to the IFR aircraft at the closest point of approach (CPA) was 
predicted to be less than 8 nmi rather than 5 nmi, and the 
primary “intent-based” conflict resolution algorithms computed 
maneuvers to remain at least 8 nmi from the IFR aircraft. If 
TBL decreased below one minute, the automation switched to a 
“state-based” conflict detection and resolution algorithm that 
used the minimum required 5 nmi separation standard (i.e., the 
3 nmi buffer was ignored). Thus, the total system attempted to 
remain outside the buffer but ultimately acted to prevent LOS 
or, in the event of LOS, to establish divergence from the 
conflict aircraft to quickly restore separation. 

The effectiveness of the buffer approach is a question of 
interest, as it is intended to provide additional time and airspace 
to maneuver for separation. The number of buffer losses, 
excluding actual and borderline LOS, is presented in Table 5. 

As the time of first alerting was reduced (by design of each 
scripted conflict), the frequency of buffer loss increased, i.e., 
the buffered airspace was put to increasing use for 
maneuvering to avoid actual LOS. Statistical analysis using the 
two-sided Fisher’s Exact Test indicated significant effects of 
TBL between the top two bins combined and the third bin (p = 
0.001), and between the third and fourth bins (p = 0.001). 
Additional testing determined that passage orientation 
geometry significantly affected the buffer loss frequency (p = 
0.004). Taking all TBL cases together, the 3 nmi buffer was a 
more effective tool in ensuring separation when the intruding 
aircraft would pass behind the AFR aircraft than pass in front. 
In further testing of geometric effects, the mean actual CPA 
was 6.18 nmi (s.d. = 0.81 nmi) for acute-angle conflicts and 
5.48 nmi (s.d. = 0.33 nmi) for obtuse-angle conflicts. A two- 
sample t-Test confirmed the encounter angle effect was 
significant (p < 0.001), indicating that the 3 nmi buffer was 
entered more frequently when aircraft converged with similar 
headings (rather than opposed headings). These results 
indicate that non-circular buffers may be equally effective at 
achieving the required separation performance while using up 
less airspace. Ultimately, in all of these buffer loss cases, the 
required minimum separation was preserved, a basic indicator 
of buffer effectiveness. 

2) Effect of Shared Intent (Exploratory Runs) 

The purpose of the exploratory runs was to compare 
operations where the trajectory intent of ground-managed IFR 


TABLE 5. Buffer Loss Results by Initial Alert Time. 


Initial Alert Time 

Buffer Losses J 

Number of Runs 

4-10 minutes 

0 

33 

2-4 minutes 

3 

32 

1-2 minutes 

13 

46 

20-60 seconds 

37 

48 

< 20 seconds 

10 

15 

Total 

63 

174 


t Excludes 3 runs in this bin with borderline loss < 0.04 nmi 


$ Excludes actual and borderline LOS 





aircraft was available to those where it was not. Other than an 
initial scripted conflict to ensure some maneuvering occurred 
during the run, conflicts occurred naturally in the complex 
airspace of Cleveland Center. 

No LOS events occurred between AFR and IFR aircraft 
throughout the exploratory runs. The frequency of conflicts 
detected by AFR aircraft during the exploratory runs is 
presented in Fig. 8. The histogram indicates, for example, that 
6 AFR flights detected exactly three conflicts during runs with 
IFR Intent Off, and 8 AFR flights detected exactly three 
conflicts during runs with IFR Intent On. Overall, runs with 
IFR Intent Off resulted in 207 conflicts, whereas runs with IFR 
Intent On resulted in 170 conflicts, an 18 percent reduction. A 
binomial test could not confirm significance of the reduction. 
As shown in the box plot, the distribution’s tail was somewhat 
reduced with IFR intent information available, indicating fewer 
cases of high-frequency detection per flight, in addition to 
overall fewer conflicts. Since each conflict alert prompts an 
AFR trajectory change to resolve it, making IFR trajectory 
intent information available to AFR aircraft promotes greater 
stability and efficiency of trajectories in the airspace. 

Out of 377 total detected conflicts, 111 were detected as 
“pop-up conflicts” with less than 4 minutes notice. Of these, 
62 were detected with IFR Intent Off, and 49 were detected 
with IFR Intent On, a 21 percent reduction and improvement. 
The frequency of pop-up conflicts detected per flight is 
presented in Fig. 9. Although higher-frequency detections 
were not substantially affected by the availability of IFR intent 
information, flights with only one pop-up conflict decreased 
and the flights with zero pop-up conflicts increased. In this 
study, conflicts detected with sufficient notice (e.g., 4-10 min.) 
could be resolved with a “strategic” FMS route modification or 
cruise altitude change. Conflicts with less notice required a 
“tactical” response using open-loop maneuvering, a deviation 
from trajectory-based operations. The flight crew procedure 
for resolving conflicts with less than four minutes time 
available was to disengage FMS guidance and maneuver 
without delay using AOP’s tactical resolution advisories. A 
total of 122 such mode switches were made, with 73 occurring 
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in IFR Intent Off runs and 49 occurring in IFR Intent On runs, 
a 33 percent reduction. The number of strategic-to-tactical 
mode switches made by the flight crews is presented in Fig. 10. 
These data show that frequent switches to tactical mode were 
reduced when intent information was available. 

Of the 207 conflicts detected with IFR Intent Off, only 15 
eventually involved a buffer loss. For IFR Intent On, only six 
of the 170 conflicts involved buffer loss. This indicates the 
buffer airspace was rarely entered in the “normal” operations 
tested in the exploratory runs. Reducing buffer size may 
therefore be feasible, allowing more efficient use of airspace 
without reducing safety. 

During group debrief sessions, the subject pilots had the 
opportunity to share and discuss their impressions of AFR 
operations and the issues tested in this experiment. Several 
pilots expressed a strong need for IFR intent information, 
specifically descent points and turns. Although opinions were 
varied, pilots generally agreed that a minimum conflict warning 
time of two minutes is required, with 5-10 minutes preferred 
for acceptability. It was also noted that ATC would not 
normally turn an IFR aircraft into conflict with an AFR aircraft, 
which will naturally limit the occurrence of late conflict 
detections. Because late detections often resulted in large 
trajectory changes and inefficient routes, it was noted that 
improvements are needed in the automation’s tactical 
resolution logic to allow more efficient maneuvers. There was 
general agreement that the AFR procedures were easy and that 
the crew coordination was effective. 

V. Conclusions 

Two human-in-the-loop simulation experiments were 
designed and conducted in a coordinated effort to explore 
function allocation of separation assurance through a mixed- 
operations concept. The en route concept included ground- 
separated aircraft and self-separating aircraft in shared airspace, 
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Figure 8. Conflicts detected for conditions of IFR Intent On and Off. 


Figure 10. Number of switches from strategic to tactical mode for 
conditions of IFR Intent On and Off. 


as well as varying levels of function allocations between 
humans and automation systems as NextGen technologies 
emerge. Simulation platforms incorporating advanced ground 
and aircraft automation tools were integrated and instantiated at 
two NASA research labs to test pilot-focused and controller- 
focused perspectives of the mixed-operations concept. 
Conclusions from the coordinated experiments are drawn with 
respect to three principal research questions. 

How does the presence of self-separating aircraft affect the 
performance of the ground-based separation system? 

To assess the impact of AFR aircraft on the performance of 
the ground-based separation assurance system, the results from 
IFR-only runs were compared to the results from mixed 
IFR/AFR runs in the controller-focused study. There were no 
significant differences between these types of operations in 
terms of separation violations, controller workload, 
acceptability, and situation awareness. Given that only a small 
subset of aircraft were conducting AFR operations, it can be 
concluded that the presence of a few self-separating aircraft 
does not impact the performance of the ground-based 
separation system. 

What are the limits of ability of self-separating aircraft to 
shoulder the burden of mixed operations? 

To determine the conditions within which self- separating 
aircraft could ensure separation from ground- controlled 
aircraft, flight crews tested a variety of conflicts with scripted 
alert timings and geometries. Using airborne automation tools 
and procedures, flight crews provided separation in all cases 
where at least one minute warning was given, regardless of 
encounter geometry. Pilots expressed a preference for at least 
five minutes notice. Two principal elements were identified to 
ensure such notice is normally available, thereby increasing 
safety, efficiency, and acceptability of mixed operations. The 
first element is airborne access to trajectory-intent information 
on planned maneuvers of traffic aircraft, either through ADS-B 
messaging or uplink from ground information systems. The 
second element is coordination between controllers and self- 
separating pilots on unplanned maneuvers of ground-controlled 
aircraft. The use of separation buffers to minimize the need for 
such coordination was effective but resulted in somewhat 
inefficient use of airspace. 

How will the implementation and maturation of NextGen 
affect the ability to operate with mixed operations? 

This question was addressed in the controller focused study 
by conducting IFR-only and mixed IFR/AFR operations in four 
different stages of NextGen. Overall, the Maximum NextGen 
stage outperformed all others by providing the same level of 
safety as the Baseline at twice the throughput and a lot less 
controller workload. In addition, only the Maximum stage 
enabled flight crews to provide their own separation from all 
other traffic with as little maneuver delay as the ground-based 
system, while all other stages required extra maneuvering. 

As advancements continue to be made in airborne and 
ground-based computing platforms, automation tools, and data- 
exchange capabilities, a variety of methods of managing 
aircraft trajectories will emerge naturally. This research effort 


has focused on determining whether multiple emerging 
function-allocation schemes for separation assurance can 
coexist in the same airspace. The approach emphasizes making 
effective use of all resources - human, automation, airborne, 
and ground-based - to achieve the most flexible and effective 
air transportation system possible to serve a highly diverse 
community of aircraft operators. 
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